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WO 99/50736 



1 

Paper Indexing Of Recordings 



PCT/US98/20593 



The present invention relates to information capture and processing, and 
more particularly to paper-based indexing of recordings. 

EP-A-495612 discloses a notetaking system based on a notepad computer 
5 with an integrated audio/video recorder, whereby a document is created or retrieved. 
As the user types on the computer's keyboard or writes with a stylus on a touchscreen, 
each character or stroke that is input by the user is invisibly time-stamped by the 
computer. The audio/video stream is continuously time-stamped during recording. To 
play a section of the recording back, the user selects part of the note and invokes a 
10 "playback selection" command. The computer then examines the time-stamp and 
winds the records toe the corresponding place in the audio/video recording, where it 
starts playing, so that the user hears and/or sees what was being recorded at the instant 
the selected text or strokes were input. (A similar system is Hewlett Packard's 
Filochart system). 

15 However, the abovementioned systems rely on the use of a dedicated 

computer with a touchscreen which electronically both captures the user's notes (or 
'indicia') at the time of recording and displays the indicia for selection by the user in a 
'playback' mode. The systems require a file to be created or retrieved before a user 
can start making notes in it or continue making notes in an already existing electronic 

20 document. By considering an example — a user being handed a set of papers 
accompanying a talk or in a meeting — it can be seen that the user cannot, using the 
aforementioned system, mark his notes or indicia on the paper he has just been given, 
but must break off from the meeting, create a new file or electronic document and set 
this up displayed on the screen of his computer so that his notes can be captured and 

25 recorded. 

Indexing audio recordings via notes taken on paper using a pressure-sensitive 
clipboard has been reported (Lisa Stifelman's Audio Notebook, MIT Media Lab ). 

The present invention provides a method carried out in an information 
processing system, for providing access to a data set, comprising: recording the data 
30 set and, concurrently therewith, recording user data, the user data being dependent 
upon user actions carried out during the recording of said data set. 
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The invention preferably provides a method carried out in an information 
processing system, for providing access to a data set, comprising: (a) providing a 
manually operable an positionable input device, the input device including a marking 
device and an image capture device intergral therewith for capturing images at or near 
5 a marking tip of the marking device, (b) providing a recording device for recording 
said data set, and means for automatically time-stamping said data set, (c) in response 
to the user, with respect to a substrate, the substrate comprising a sheet medium and 
coded machine-readable markings formed on the sheet medium, said markings being 
formed so as to define a plurality of zones on the substrate, each zone containing 

10 machine readable first markings defining a page identification code, the page 
identification code being unique to the substrate, (A) positioning the input device at a 
location on the substrate, or (B) causing the input device to perform a movement 
across the substrate, capturing images of the substrate within the field of view of said 
image capture device while the input device is positioned at said location or during 

1 5 said movement, (d) from the image captured in step (c), determining the page location 
codes within the or each zone of the substrate within said images, and automatically 
time-stamping said page-location codes upon their generation, said data set and page 
location codes being time-stamped with a common time base. 

The invention further provides a method carried out in an information 

20 processing system for outputting randomly accessible parts of a recorded data set, the 
data set having been recorded according to the above methods, comprising: storing 
association data defining a mapping between user data and respective parts of said 
data set, and in response to a user input defining a user action selection, said user 
action selection defining a selected one of the user actions, using the user data 

25 corresponding to the selected user action, outputting the part(s) of the data set 
corresponding to said user action. 

This invention preferably provides a method carried out in an information 
processing system for outputting randomly accessible parts of a recorded data set, the 
data set having been recorded according to the above methods, comprising: (k) 

30 providing a manually operable an positionable input device, the input device 
including an image capture device intergral therewith for capturing images at or near a 
tip of the input device, and providing an output device for outputting said data set, (1) 
in response to the user, with respect to a substrate, the substrate comprising a sheet 
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medium and coded machine-readable markings formed on the sheet medium, said 
markings being formed so as to define a plurality of zones on the substrate, each zone 
containing machine readable first markings defining a page identification code, the 
page identification code being unique to the substrate, (A) positioning the input device 
5 at a location on the substrate, or (B) causing the input device to perform a movement 
across the substrate, capturing images of the substrate within the field of view of said 
image capture device while the input device is positioned at said location or during 
said movement, (m) from the image captured in step (1), determining the page location 
codes within the or each zone of the substrate within said images, (n) from the page 

10 location code(s) determined in step (m), deriving the time-stamp(s) associated 
therewith, (o) from the time-stamp(s) derived in step (n), outputting corresponding 
part(s) of said data said using said output device. 

The invention further provides a programmable information processing 
system when suitably programmed for carrying out the method of any of the 

1 5 preceding claims, the system including a processor, memory, a manually operable an 
positionable input device, the input device including a marking device and an image 
capture device intergral therewith for capturing images at or near a marking tip of the 
marking device, a recording device for recording said data set, means for 
automatically time-stamping said data set, and an output device for outputting said 

20 data set, the processor being operable in conjunction with the memory, input device, 
recording device, automatic time-stamping means, and output device, to execute 
instructions corresponding to said methods. 

The invention is concerned with ways of indexing recordings with notes 
written on paper, in a number of embodiments. A preferred embodiment is via coded 

25 substrates (see GB application 98 (applicants' ref R/98003/JDR), which 

provides functionalities not provided by other embodiments (because they are 
standard functionalities of the coded substrates). 

Thus, the invention may be embodied in a system for taking notes on paper 
during a recording (typically audio or video) and using them as an index to control 

30 playback of the recording. Extensions may allow the user to take notes either on 
paper or with an electronic device, and to control playback either with a paper copy or 
an electronic version of the notes. 
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The user makes a note on paper with a combined pen/camera or with pen or 
pencil; if with pen or pencil, either subsequently scanning in the marks or writing 
under the eye of a camera. An identifying feature of the note is computed. The 
feature is stored with the current time-stamp of the recording. This process may be 
5 repeated. A note may be subsequently selected. Its identifying feature is computed 
and matched against the set of stored features and the time-stamp associated with the 
matching element is recovered, and used to initiate replay of the associated portion of 
the recording. 

A preferred embodiment uses substrates, e.g. paper, or documents produced 
10 therefrom, which include visible or invisible coded markings identifiying the substrate 

and preferably locations or zones within it. This marking scheme in turn preferably 

uses Xerox DataGlyphs. 

An advantage is that the invention allows handwritten notes on 'standard' 

paper or other conventional media to index and control playback of recordings. 
15 In an alternative, handwritten paper notes may be timestamped without 

respect to an explicit recording (examples: ships log, news reporting). Handwritten 

paper notes may be time-correlated with instrumentation logs (examples: laboratory 

experiments, telescope parameters, surgery-room parameters, meteorological 

conditions). A variant would not use time, but directly associate parameter values 
20 (such as the telescope's position) with the note. In this case a specialized application 

would be needed to interface to the instrument. 

In implementing the techniques according to the invention, the identifier 

routing techniques described in GB application 98 (applicants' ref 

R/98005/JDR) may be employed. 
25 Embodiments of the invention will now be described, by way of example, 

with reference to the accompanying drawings, in which: 

Figure 1 illustrates the components of a printed document as printed on a 

coded substrate; 

Figure 2 shows a sample of zones, and the disposition of machine readable 
30 data, on a coded substrate; 

Figures 3 and 4 show how digital data is encoded in the zones illustrated in 

Fig. 2; 
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Figure 5 schematically illustrates an embodiment of a pointer which may be 
used in implementing the invention; 

Figure 6 shows a configuration for passing page identification codes and/or 
page location codes from the pointer of Fig. 5 to a network computer, in accordance 
5 with an embodiment of the invention; 

Figure 7 illustrates an overview of a system suitable for implementing an 
embodiment of the invention; 

Figure 8 is a schematic illustration of a first processing operation, in 
recording according to one embodiment of the invention, performed in the processor 
10 device of Fig. 7; 

Figure 9 shows a schematic illustration of a second processing operation, in 
recording according to one embodiment of the invention, performed in the processor 
device of Fig. 7; 

Figure 10 is a schematic illustration of a third processing operation, in 
15 recording according to one embodiment of the invention, performed in the processor 
device of Fig. 7; 

Figure 11 is a schematic illustration of a fourth processing operation, in 
recording according to one embodiment of the invention, performed in the processor 
device of Fig. 7; 

20 Figure 12 shows a schematic illustration of a first processing operation, in 

retrieval according to one embodiment of the invention, performed in the processor 
device of Fig. 7; 

Figure 13 is a schematic illustration of a second processing operation, in 
retrieval according to one embodiment of the invention, performed in the processor 
25 device of Fig. 7; 

Figure 14 illustrates an overview of a system suitable for implementing an 
embodiment of the invention permitting paper annotaion for electronic retrieval; 

Figure 15 is a schematic illustration of a first processing operation, in 
recording according to one embodiment of the invention, performed in the processor 
30 device of Fig. 14; 

Figure 16 shows a schematic illustration of a second processing operation, in 
recording according to one embodiment of the invention, performed in the processor 
device of Fig. 14; 
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1. General features 

The invention concerns means of taking notes on paper during a recording 
(or during playback of a recording), and using them as indices into the recording for 
5 controlling playback. Two means are described, one in which the position of the 
annotation is used to identify it, and one in which the shape of the annotation 
identifies it. 

An alternative form involves a facility to take notes using an electronic 
device (keyboard, palmtop, ...), but to print them and control playback using the 
10 printed copy, or to take notes on paper but control playback through an electronic 
device. 

2. Paper Annotation and Retrieval 

Elements of embodiments of the invention are: 
Annotation 

During a recording session, or while a recording is being played: 
the user takes notes on paper; 

for each note, the system stores some identifying feature of the note with the 
time-stamp of the recording (as of the moment the note was taken). 

Selection and retrieval 
In order to control playback of a recording: 
the user selects a note from the paper; 

the system determines the identifying feature of the note, looks it up among 
those it has stored, and retrieves the associated time-stamp (which is subsequently 
used to play the recording starting at the position identified by the time-stamp, in 
accordance with prior art). 

An overview of the system is shown in Fig. 7. The components are: 
30 1 . A sheet of paper, on which a person takes notes during a recording session 

or playback of a recording in order to indexing the recording, or selects a note in order 
to control playback of an indexed recording; 

2. A camera, which delivers an image of the paper to 
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3. A processing system, which produces from that image 

4. an Annotation ID; 

5. A recording or playback device, which produces the current Timestamp 
during a recording session or playback of a recording when indexing the recording, or 

5 consumes a timestamp and plays the recording beginning at that timestamp, when 
playback is being controlled; 

6. a Timestamp, 

7. a table, in which pairs consisting of an Annotation ID and a Timestamp are 
stored during indexing or looked up during control of playback. 

10 For simplicity of exposition, assume that there is a button available to the 

user. She clicks it to signal that she's starting an annotation. The time-stamp of the 
recording as of when she clicks is selected to be associated with the annotation. 
Presumably she clicks in reaction to an interesting event in the recording session. 
There will be a lag due to her reaction time, so when she views the material associated 

15 with the annotation, she probably wants to start playback of the video at a point prior 
to the selected time-stamp. The system should provide a parameter to let the user 
offset actual playback from the time-stamp point. When she's done, she signals the 
fact by clicking again. 

There is a natural place for this button in some variants and not in others; in 

20 addition, in some variants the beginning of an annotation may be automatically 
detectable. These are not the most essential elements of the invention, so we don't 
discuss them further. 



2.1 Identifying Annotations by Position 

25 In this embodiment, the identifying feature of an annotation is its position on 

the page. The position of the marks the user makes on the page are used to define a 
region that bounds the entire annotation. This can be done in many ways, and the 
exact method used is not important. A representation of the region identifies the 
annotation and is stored with the time-stamp. 

30 You select a note on the page by pointing at some part of it. The pointed-at 

position is checked against the defined regions, and the time-stamp of the one it lies 
within is retrieved. 
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Coded substrates provide one mechanism for recovering positional 
information, and an externally mounted camera provides another. 

2.1.1 Coded substrates 

Figure 1 illustrates the components of a pointed document as printed on a 
coded substrate. The printed document 102 comprises a layer 104 of printed visible 
(human-readable) information printed on a coded substrate 106. The coded substrate 
106 in turn comprises a layer 108 of visible or invisible machine readable markings 
printed on a sheet medium 110 (e.g. paper). 

Figure 2 shows a sample of zones, and the disposition of machine readable 
data, on a coded substrate. Each zone or cell 202 includes a border 204 and an 
orientation marker 206. A first set of markings 208 over part of the interior of the cell 
202 are encoded representations of the page-id, while a second set of markings 210 
over a (smaller) part of the interior of the cell 202 are encoded representations of the 
localisation (page-loc) - uniquely defining the position of the cell 202 within the page. 

Figures 3 and 4 show how digital data is encoded in the zones illustrated in 
Fig. 2. Fig. 3 shows the binary data, i.e. 47 bits of page-id in the upper section 302 
(the bit stream wraps at the cell border 204), and 16 its of page localisation data (loc) 
in the lower section 304). The page-id code denotes 108429159095492 = 
629DA1 82DCC4 (hexadecimal) 
1 1000101001 1 101 10100001 100000101 101 1 1001 10001000001000000010101 
(binary; to make the wrapping explicit). In the 16 bit loc code in section 304, there are 
8 bits for the X co-ordinate and 8 bits for the Y co-ordinate. Thus, for the cell (zone) 
shown, its position is 16,21 on the substrate. 

Fig. 4 shows the same data as in Fig. 3, but represented by Data Glyph 
markings. Encoding using data glyphs and the retrieval of data therefrom is discussed 
further in US-A-5,486,686, EP-A-469864, and the abovementioned GB application 
(ref.R/98003/JDR). Here, there is a first set of glyphs (markings) in upper section 402 
and a second set in lower section 404, the two sets of glyphs being encoded 
representations of page-id and loc codes. 

Figure 5 schematically illustrates an embodiment of a pointer which may be 
used in implementing the invention. The pointer 502 comprises a marking device 504 
(which may be a pen or any other marking device suitable for making marks which 
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are visible to a user), and an image capture device 506. In use, whether or not the user 
is making marks using the marking device 504, the image capture device 506 is able 
to capture images of an area A of a document 508. (For the sake of illustration, the 
sizes of these elements are exaggerated - e.g. in practice, the area A may be much 
5 closer to the tip 505 of the marking device 504 than appears). In certain 
embodiments, the marking device 504 may be omitted. 

The document 508 may be a 'blank' coded substrate, or such a substrate 
having human-readable information printed thereon. 

Figure 6 shows a configuration for passing page identification codes and/or 

10 page location codes from the pointer of Fig. 5 to a network computer, in accordance 
with an embodiment of the invention. The image capture device (e.g. CCD camera) 
506 is coupled by wired or wireless (e.g. IR or RF) link to processing device 602 and 
in use provides image data defining capture images to the processing device 602. The 
operative elements of the processing device 602 are a grame grabber circuit 604, 

15 image decoding software 606, and a CPU 608, which are known in the art. (In certain 
embodiments, the camera 506 and processing device 602 may be combined into an 
integral handheld unit). In use, the processing device 602 extracts from the image 
data the corresponding page-id and page-location data (<pid, loc>) and communicates 
them in a wired or wireless fashion to a local device (here, a network computer 610, 

20 which is linked to the network (intranet, internet) in a known manner). The computer 
610 has its own unique network address, but need not have any information output 
device (e.g. display screen, printer). 

The user writes on a coded substrate with the pointer (a hand-held 
pen/camera combination). 

25 As she writes, the positions of the pointer while it is in contact with the page 

are recovered. (This is one of the functionalities of the coded substrates; it is enabled 
by a substrate of glyphs on the page.) 

The user selects an annotation by clicking the pointer on some part of the 
annotation; the position of the pointer is recovered (again, as a natural product of the 

30 use of Intelligent Paper). 

The use of coded substrates also — for free — automatically launches the 
annotation/playback application when you click on the page, and automatically 
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locates the set of note identifiers and time-stamps for the page in use, wherever they're 
stored. 

Figure 8 shows the operations performed by the processor of Fig. 7. 

Here, a series of images are recovered by the camera. Processing these in the 
5 standard manner for coded substrates produces a series of positions. A bounding 
region (for example, a bounding rectangle) is computed and serves as the annotation 
identifier. 



2.1.2 Under the Camera 

10 With coded substrates, the camera lies in a hand-held device. In the systems 

of EP-A-622,722 and EP-A-840,199, the camera is in a fixed position over the desk. 
This invention does not use the full set of concepts of those European patent 
applications, but employs the mounted camera concept from those works. 

The user annotates a sheet of paper with pen or pencil. The page is within 

15 the field of view of the camera. It has registration marks on it which allow distinct 
images of it to be registered properly with each other in the case that the page is 
moved. It may also have identification marks to distinguish it from other sheets of 
paper. 

For simplicity's sake, assume that the entire page is unobscured immediately 
20 before and after the annotation is made. The last image of the page taken before the 
annotation is made is subtracted from the first image taken after, isolating the marks 
comprising the annotation. The positions of the marks relative to the page frame are 
recovered. 

The user selects an annotation by pointing at some part of the annotation. 
25 Pointing is recognized by the image processing system. (Selection techniques are 
discussed in EP-A-622,722 and EP-A-840,199). 

Figure 9 shows the operations performed by the processor of Fig. 7 in this 
embodiment. 

Here, two images are recovered by the camera and calibrated using the 
30 identifying marks in the comers. Subtracting the first from the second produces a 
third image containing only the current annotation.. A region bounding the annotation 
is computed and serves as the annotation identifier. 
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2.2 Identifying Annotations on Special-Purpose Pages 

In this embodiment, the identifying feature of the annotation is a number that 
is encoded in marks on the paper ahead of time. For annotation, the user scans one of 
the marks, then writes on the paper next to it. The scanned mark is decoded, and the 
5 number is stored with the time-stamp of the recording. 

For selection and retrieval, the user selects a note by scanning the mark. It is 
decoded into a number, and the time-stamp associated with that number is retrieved. 

Glyphs are a natural choice for encoding the numbers.The additional 
functionalities of coded sybstrates can be provided by using such substrates without 
10 localisation codes pages as the special-purpose pages. 

Figure 10 shows the operations performed by the processor of Fig. 7 in this 
embodiment. 

Here, an image is recovered by the camera. Processing the image and 
decoding the glyphs in the standard manner (using one of Xerox's glyph toolkits, 
15 either SmartPaper, DAE, or DataGlyphs) produces a number, which serves as the 
annotation identifier. 



2.3 Identifying Annotations by Shape Processing 

In this embodiment, the annotations are written on the page with pen or 
20 pencil and subsequently scanned. The shape of the annotation identifies the 
annotation. This is similar to a method used in PaperLink to implement paper 
hyperlinks. An annotation is selected to control playback by scanning it. 

In slightly more detail: when something interesting happens in the recording, 
the user clicks to select the current point, and the system picks up the time-stamp. 
25 She writes a note. When she's done writing, she scans what she has written with a 
hand-held scanner. When she's using the system to control playback of the recording, 
she scans a note, the system retrieves the associated time-stamp and initiates playback 
of the recording at that point. 

The annotation identifier (a set of features extracted from the scanned image) 
30 and time-stamp are stored for each note. When the user selects an annotation, it is 
compared to the stored images (or features are extracted and compared to the stored 
feature sets). The one that matches best is identified, and its time-stamp is retrieved. 
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Matching should be tractable and robust because there is a limited set of annotation 
identifiers to match against. 

The user need not scan the entire note, either when taking notes or selecting 
them; of course, she does have to make sure that what she scans for selection is pretty 
5 much what she scanned at the point of making the annotation. 

Figure 1 1 shows the operations performed by the processor of Fig. 7 in which 
embodiment. 

Here, an image is recovered by the camera. Standard image processing of it 
produces a set of image features, which serves as the annotation identifier. 

10 

2.4 Multiple Pages 

A given page of notes is associated with a particular recording. Under the 
assumption that multiple pages of notes are taken, referring to multiple recordings, 
storing this association with the note/time-stamp mapping, and identifying this 

1 5 aggregate structure via a page identifier becomes useful. Intelligent Paper incorporates 
page identification, and as mentioned, the mounted-camera embodiment provides for 
page identification. Pre-marked pages can also provide page identification by keeping 
the numbers distinct across distinct pages. For the shape-processing embodiment, a 
set of image features can serve to identify a page. The user can make an arbitrary 

20 mark on the page to identify it, then scan the mark. Collisions are possible but can be 
detected. 

2.5 Other Input Devices 

Audio and video recordings are in widespread use, but this technique can 
25 index any data stream whose values change over time. For example, if the user had a 
GPS device whose values were accessible to the system, notes could be associated 
with locations. 

3. Electronic Annotation and Paper Retrieval 

30 Here the user takes notes using an electronic device, and the notes are printed 

so that playback can be controlled from a paper copy of the notes. As in the previous 
case, the system stores for each note an identifying feature of the note along with the 
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time-stamp. The choice of feature and process of computing the feature depends on 
the nature of the input. 

3.1 ASCII and OCR 

5 If the annotations are input via a keyboard we have them in the form of ascii 

text. They can be printed in any fashion. An annotation is selected from the printed 
page by scanning it with a hand-held scanner. The scanned text is recovered using 
OCR. The annotation containing the most closely matching text is identified, and its 
associated time-stamp is retrieved. 
10 OCR accuracy can be improved by storing font information at the point of 

printing the annotations. 

Figure 1 2 shows the operations performed by the processor of Fig. 7 in this 
embodiment. 

Here, an image is recovered by the camera. Optical character recognition is 
1 5 performed, producing a text. This text is matched against the set of stored annotation 
identifiers (which are the text of the annotation, in this embodiment) and the closest 
match serves as the annotation identifier. 

3.2 Annotation ID Marks 

20 When the annotations are printed, an identifying mark can be printed with 

each note (in the margin, for instance). The mark is used as the key in time-stamp 
retrieval, or if the mark represents a number, the number is used as the key. The user 
selects the note by hand-scanning the mark, which is recognized using techniques 
appropriate to the kind of mark. Possible marks are glyphs, bar codes, numbers or 

25 symbols in OCR fonts, icons, or text. 

Figure 1 3 shows the operations performed by the processor of Fig. 7 in this 
embodiment. This is similar to Fig. 10. 

3.3 Compatibility 

30 In a system designed for taking notes electronically, one of the two above- 

described methods is appropriate for selecting them from paper to control playback. 
But a system designed for taking notes on paper could be extended to allow notes to 
be taken electronically and printed. In this case, selection from paper should use the 
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same mechanism as if the notes had been taken on paper; the annotation identifiers are 
defined by the system for taking notes on paper. 

4. Paper Annotation and Electronic Retrieval 

5 The above-described systems for paper annotation can be augmented to allow 

electronic retrieval. 

An overview of the system is shown in Fig. 14. The elements are the same as 
in Fig. 7, with one addition: the images are stored in the table with the Annotation IDs 
and Timestamps. This allows a composite image (a) identical to the paper that the 

10 notes were written on to be constructed and displayed for the user on a screen. 

In the positional case, i.e. with localisation code (loc), when the annotations 
are made, the image of the page must be stored. The final stored image is displayed to 
the user. The mouse position (relative to the page image frame) is used just as the 
position on the paper would be for retrieval purposes. 

15 In the hand-scanning case, a display can be built from the various scanned 

notes. When the display is built, the time-stamps must be associated with regions of 
the display, so that the mouse can be used for retrieval. 

Figure 15 shows the operations performed by the processor of Fig. 14 for 
positional-based retrieval. 

20 This Figure shows the elements as used in retrieval (using annotations to 

control playback of an indexed recording) in the positional cases (coded substrates 
and Over-the-desk Camera). The composite image (a) described above is displayed 
on a screen to the user. The user selects an annotation with the mouse by placing it on 
some part of the annotation. Because regions in image (a) corresponds to regions on 

25 the original paper sheet the notes were written on, the mouse coordinates identify one 
of the regions serving as annotation identifiers. The timestamp stored with the 
annotation identifiers is retrieved and used for playback. 

Figure 16 shows the elements as used in retrieval in the hand-scanning case. 
Timestamps are associated directly with regions of the display when the display is 

30 built, so the mouse position when the user selects an annotation identifies a timestamp 
directly. This timestamp is retrieved and used for playback. 
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CLAIMS 

1 . A method for providing access to a stored item of data comprising: 

using a manually positionable scanner to scan one or more regions of a sheet and to 
provide a scan signal; the sheet bearing machine-readable markings; the markings 
defining two or more zones of the sheet; the markings within each zone indicating a 
position of the zone within the sheet; 

using the scan signal to obtain position data indicating one or more positions indicated 
by the machine readable markings; 

encoding one or more positions indicated by the position data with time information 
to obtain encoded position-time data; and 

associating time data indicating the time information with the stored item of data to 
provide access to the stored item of data using the encoded position-time data. 
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